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We prove limit theorems for rescaled occupation time fluctuations of a {d, a, /3)-branching particle 
system (particles moving in according to a spherically symmetric a-stable Levy process, (1 -|- /?)- 
branching, < /3 < 1, uniform Poisson initial state), in the cases of critical dimension, d = a(l-|- /?)//?, 
and large dimensions, d > a{l + /?)//?. The fluctuation processes are continuous but their limits are 
stable processes with independent increments, which have jumps. The convergence is in the sense of 
finite-dimensional distributions, and also of space-time random fields (tightness does not hold in the 
usual Skorohod topology). The results are in sharp contrast with those for intermediate dimensions, 
a/(3 < d < d{l + I3)/I3, where the limit process is continuous and has long range dependence (this case 
is studied by Bojdecki et al, 2005). The limit process is measure- valued for the critical dimension, and 
5'(M'^)-valued for large dimensions. We also raise some questions of interpretation of the different types 
of dimension-dependent results obtained in the present and previous papers in terms of properties of 
the particle system. 

Key words: branching particle system, critical and large dimensions, limit theorem, occupation time 
fluctuation, stable process. 
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1. Introduction 



We consider the {d, a, /3)-branching particle system, which consists of particles evolving independently 
in according to a spherically symmetric a-stable Levy process (called standard ct-stable process 
henceforth) and a (1 + /3)-branching law, < /3 < 1. This law has generating function 

s+^—^^^, 0<s<l, (1.1) 

is critical, and belongs to the domain of attraction of a stable law with exponent 1 + /? (the case /3 = 1 
corresponds to binary branching). The particle lifetime distribution is exponential with parameter 
V (this parameter is not particularly relevant in this paper, but we keep it for consistency with our 
previous papers, Bojdccki et al 2004, 2006a, 2006b, 2005, where sometimes it plays a role). We assume 
that the system starts off at time from a Poisson random field with intensity measure A (Lebesgue 
measure) . 

The {d, a, /3)-branching particle system and its associated superprocess have been widely studied 
(see Dawson, 1993, Dawson et al, 1989, Dawson and Perkins, 1991, Etheridge, 2000, Fleischmann and 
Gartner, 1986, Gorostiza and Wakolbinger. 1991, Iscoe, 1986, Meleard and Roelly, 1990, for some of 
the early results). Although this spatial branching system is special, the different types of behaviors 
it exhibits depending on relationships between d,a and /? (e.g. the persistence/extinction dichotomy, 
Gorostiza and Wakolbinger, 1991) make it an interesting test case. 

Let N = {Nt)t>o denote the empirical measure process of the system, i.e., Nt{A) is the number of 
particles in the set A C at time t. The rescaled occupation time fluctuation process is defined by 

Xrit) = — / (iV, - X)ds = — / {Nts - X)ds, t > 0, (1.2) 

Jo Jo 

where Ft is a suitable norming for convergence and T is the scaling parameter which accelerates time 
and tends to oo. Note that ENg = A for all s > 0, due to the initial Poisson condition, the criticality 
of the branching and the a-stable motion. 

The process Xt exhibits different asymptotic behaviors as T ^ oo, depending on relationships 
between the parameters d,a,f3. For intermediate dimensions, a/ (3 < d < a{l + (3)/ (3, and Ft = 
ji(2-/3--/3)/(n-/3)^ ^j^g limit process has the form KX^, where K is a constant and ^ = {^t)t>o is a 
continuous, self-similar, stable process which has long range dependence (Bojdecki et al, 2005). In 
the present paper we consider the cases of critical dimension, d = a(l + f3)/f3, and large dimensions, 
d > q(1 + /5) //?. We prove limit theorems for Xt described briefly as follows. For the critical dimension 
and Ft = (T\ogT)^^^^~^^\ the limit has the form KX^, where K is a constant and ^ = (^t)t>o is a 
real standard (1 -|- /3)-stable Levy process totally skewed to the right. For large dimensions and 
Ft = T^/(^+l3\ the limit is an 5'(R )-valued stable process with stationary independent increments 
{S'{W) is the space of tempered distributions, dual of the space 5(M ) of smooth rapidly decreasing 
functions). These two limit processes have jumps; recall that a continuous process with independent 
increments is necessarily Gaussian (Kallenberg, 2002, Theorem 13.4); this is in fact the case for /3 = 1 
(Bojdecki et al, 2006b). 

Clearly, since the fluctuation processes are continuous and the limit processes have jumps, there 
cannot be convergence in the Skorohod space D{[0, 1],5'(M'*)) with the usual Ji-topology. We prove 
convergence of finite-dimensional distributions and also in the sense of space-time random fields (Bo- 
jdecki et al, 1986). We do not know if tightness holds in a weaker topology, e.g., the Mi-topology 
(Skorohod, 1956), or the S'-topology (Jakubowski, 1997). 

The main observations regarding the results for critical and large dimensions are the striking fact 
that the fluctuation processes Xt, which are continuous, develop jumps in the limit as T ^ oo, and 
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that the Hmit processes have independent increments. These properties are in sharp contrast with 
those for intermediate dimensions (Bojdecki et al, 2005), where, moreover, convergence takes place in 
the space of continuous functions C([0, 

In Bojdecki et al (2006a, 2006b) we proved functional limit theorems for the system with (3 = 1, 
which corresponds to finite variance (binary) branching, for intermediate, critical and large dimensions. 
In that case all the limits are continuous. Another relevant paper is Birkner and Zahle (2005), where 
functional limit theorems are proved for fluctuations of the occupation time of the origin for a system 
of critical binary branching random walks on the d-dimensional lattice, d > 3, with methods different 
from ours; the results for initial Poisson state arc parallel to those of Bojdecki et al (2006a, 2006b) and 
and in addition the equilibrium case is treated. In contrast with Birkner and Zahle (2005), we have also 
investigated the spatial structure in our case. For the (d, a, /?)-branching particle system with d = a/l3, 
there is a functional ergodic theorem (Talarczyk, 2005). The present paper has partial origins in Iscoe 
(1986), where the occupation time of the (d, a, /3)-superprocess is studied. The occupation times of 
the {d, a, /?)-branching particle system and the (d, a, /?)-superprocess have analogous properties, but 
the particle system is technically more involved. 

The methods of proof in this paper are similar to those in Bojdecki ct al (2006a, 2006b, 2005), 
with some different technical complexities; the Fourier transform tools used before for /3 = 1 are not 
applicable with (3 < 1. We refer to Bojdecki et al (2005) for some technical points. We remark that the 
critical dimension is more difficult to deal with than the large dimensions, although the limit process 
is simpler. 

Additive processes in 5'(]R'^) and other nuclear spaces are an interesting subject on its own right 
(e.g. Ito, 1980, Perez- Abreu et al, 2005, Ustunel, 1984). This article shows that processes of that type 
actually arise in physical models. 

Section 2 contains the results, and Section 3 the proofs. In Section 4 we make some comments 
and raise questions of interpretation of the results for the intermediate, critical and large dimensions 
in terms of the particle system; these are questions that require further research. 



2. Results 

The following notation will be used. ( , ) denotes pairing of spaces in duality (e.g., 5'(]R'^) and 
5(M'')). II lip stands for LP -norm, and it will be clear from the context whether the underlying space 
is R*^ or an interval of R. Constants are written C, Ci, . . . , with possible dependencies in parenthesis. 
=> denotes convergence in law in appropriate spaces, and convergence of processes in the sense of 
finite-dimensional distributions is denoted 

/ 

We also need another, less known, notion of convergence, related to space-time random fields, 
introduced in Bojdecki et al (1986). To any stochastic process X = {X(t))t>o with paths in the 
Skorohod space of cadlag functions D(R_|_, 5'(M^)) and any r > 0, there corresponds an 5'(M°'^^)- 
valued random element X defined by 

{X,^)= r{Xit),^;t))dt, $g5(R'^+1). (2.1) 
Jo 

Definition 2.1 Let X, X^, n = 1, 2, . . . , be 5'(R"')-valued cadlag processes. We say that the laws of 
Xji converge to X in the space-time, or integral, sense (denoted Xn =^ X) if for each r > 0, 

i 

X„ X as n — ^ oo. 
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=^ convergence, resembles convergence but none of them implies the other. Each one of these 

i f 

convergences together with tightness on D([0, r], 5'(M'^)) implies convergence in law in D{[0, r],(S'(M'^)). 
Conversely, weak functional convergence implies =>. 

i 

Let pt denote the transition density of the standard a-stable process on W^, and % the correspond- 
ing semigroup, i.e., %if = pt* ip. Denote the potential operator 

G^{x) = TMx)dt = C,,d ^-f^^dy, (2.2) 

where Ca,d = r(^^)(2"7r2r(^))^-'^ (we always assume d > a). 

The main result of this paper is the following theorem for the occupation time fluctuation process 
of the (d, a,/3)-branching particle system defined by (1.2). 



Theorem 2.2 Assume < /? < 1. 
(a) Let 



and Ft = T^+0. Then 



Xt^^X and Xt=^X as T ^ oo, 
i f 



where X is an S^W^) -valued (1 + P)-stable process with stationary independent increments whose 
distribution is determined by 

Eexp{i{X{t),^)} =exp!^-K^+^t J JGif{x)\^+^ (^1 - i{sgnG^ix))ian^{l + P)^dx^ , (2.4) 

(p eS(W^),t>0, 



where 



(b) Let 



1 

V TT \ 1+/3 



rf=2li±ia (2.5) 



and Ft = (T log T) . Then 



Xt^^KiX^ and Xt^^KiX$, as T — > oo, 

i f 

where ^ is a real (1 + (3) -stable process with stationary independent increments whose distribution is 
determined by 

£;exp{iz^t} = exp|-i|z|^+'^(^l-i(sgnz)tan|(l + /?))|, zeR,t>0, (2.6) 
and ^ 



{x)dr ) pi{x)dx 
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Remarks 2.3 (1) Existence of a family of finite-dimensional distributions of the process X described 
in Theorem 2.2 (a) follows, e.g., from the proof. Then the existence of the 5'(M'^)-process X itself is 
a consequence of classical properties of nuclear spaces (regularization theorem. Ito, 1980). 

(2) The process in Theorem 2.2 (b) is the standard (1 + /3)-stable Levy process totally skewed to 
the right. 

(3) The weak limit of (XT'(t), ip) in Theorem 2.2 (a) coincides with the result obtained by Iscoe (1986, 
Theorem 5.6) for the (d, a, /?)-superprocess. 

(4) For /? = 1 and large dimensions, in Theorem 2.2 (a) of Bojdecki et al (2006b) there is an additional 
term in the limit process which comes from the free motion of the particles (i.e., without branching). 
The limit is an 5'(M'^)-Wiener process, which is the sum of two independent 5' (M"^)- Wiener processes. 
The additional term does not appear with (3 < 1 because in this case the branching produces larger 
fluctuations (of order T'^l^^+f^)) than those of the free motion (of order T^/^). 

It is clear that weak functional convergence of Xt does not hold (with Ji-topology). Neverthe- 
less, it turns out that the family {Xt}t posesses a property which is "not far" from tightness in 
(7([0, r], 5'(M'^)). Namely, we have the following proposition. 

Proposition 2.4 Assume d > and let Ft be as in Theorem 2.2. Then for any f £ S{M ) and 

T>0, 

P{\{XT{t2),v) - {XT{h),^)\ >5)< ^|i2-ti| (2.7) 
for all tiM e [0,r], allT >2 and allO <6 <1. 



3. Proofs 

Before the proof of Theorem 12.21 we state a simple lemma regarding the operator G given by (2.2), 
which will be used frequently (see, e.g., Iscoe, 1986, Lemma 5.3). 

Lemma 3.1 If ip is a measurable function on such that 

sup (1 + |x|^)|(/?(x)| < oo 

for some p > d, then 

sup(l + |x|'^~")|Gv9(x)| < oo. 

Proof of Theorem 2.2. Without loss of generality we assume r = 1. 

We need the form of the Laplace transform Eex.p{ — {XT,^)} for $ G 5(M'^'''^), $ > 0, where Xt 
is defined by (EU). 

Denote ^ 

^{x,t) = ^{x,r)dr, ^iT{x,t) = ^^{x,^, (3.1) 

and define 



VT{x,t) = l-^exp|-^ {N^,-^t{-,T -t-r)dr^, 



(3.2) 



where A'^ is the the empirical measure of the branching system started from a single particle at 
X. Identically as in Bojdecki et al (2006a) (see also Bojdecki et al, 2006b, Lemma 3.1), using the 
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Feynman-Kac formula and the form of the generating function of the branching law given by (jLlj) , it 
can be shown that vt satisfies 



VT{x,t) 



t-r 



^T{;T-r){l-VT{;r)) 



1 + /?"^^ 



{x)dr, 0<t<T, 



and 



Eexp{-{XT,m = Eexp\ / {Nr,^Ti;r))dr 



exp 



^rix, T — r)vT{x, r)drdx 



V 



^t{x^ r)drdx 

T 

v}p^^{x, r)drdx 



(3.3) 

(3.4) 
(3.5) 

(3.6) 
(3.7) 



1 + /3 Jo 

We will use frequently the following estimates for vt- 

< VT{x,t) < 1, 

by (jSH, and 

VT{x,t)< [ Tt_r'^Ti-,T -r){x)dr, 
Jo 

since 1 - e'"' < x, x > 0, and E{N^, (p) = Ttip{x). (((S21) also follows from (jHSJ and (jHSI)). 

For the convergence of finite-dimensional distributions we need also the corresponding Laplace 
transform. For tpi, ip2, . . . ,Lpk G 5(R'^), all Lpj > 0, and Q < ti < t2 < • ■ ■ < < 1., '\i '\s easy to see 
that E'expj— Yl^j=i{-^T{'tj)i^j)} has the form 1)3. 4|1 with 

k 



(3.8) 



Moreover, approximating \I' by smooth functions in (|3.3() . we obtain that for this Laplace transform 
(|3.5|) also holds with a corresponding vt given by (|3.2|) . 

It will be convenient to write the right hand side of H3.5() in the form 



exp|/i(r) + 



V 



1 + 13 



{h{T)-h{T)) 



where 



h{T) 
HT) 
h{T) 



r 

/ "^t{x,T — r)vT{x,r)drdx, 
Jo 

[ ( [ Tr-u'i>T{-,T -U){x)du) 

Jo \Jo / 

[ \( [ %-u^t{-,T -u){x)du 

"Jo [\Jo 



1+/3 

drdx, 

1+/3 



Vrp [x, r) 



drdx. 



(3.9) 

(3.10) 
(3.11) 
(3.12) 



We now prove convergence in case (a). 
Firstly, we want to show that 



lim Sexp-(XT,$) = exp|— ^ / / (G'^(-,r)(x))^+'^ drdx\ (3.13) 

+ P Jr'^Jo J 
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for $ G 5(M'^+^), $ > 0, and ^' given by p.lf) . To simplify notation we consider $ of the form 

$(x,t) = ipeS{W^), ipeS{W), ip,ip>0. (3.14) 

Denote 



= ^{r)dr, XT{t) = x[j y ^t{x) = jr^{x). (3.15) 
Using we will show that 

hiT) ^ 0, (3.16) 

hiT) [ [ {G^{x)x{r)Y^^drdx, (3.17) 

/3(T) ^ 0, (3.18) 

as T ^ oo. 

By (|3.7j) , (|3.15j) and boundedness of x we have 



ip{x)dudrdx 



T 

T f 

< C — — I ip{x)G(p{x)dx, 

by H2.2p . G(p is bounded by Lemma 3.1, hence 1)3. 16() follows since /? < 1. 
Next, we use (|3.15|) and make obvious substitutions to obtain 

1/ r^a-r) / u\ 



hiT) = J^J^ l^J^ TMx)x\^r + -jduj drdx. (3.19) 
It is clear that 

j 'Tu^{x)x\r + —jdu = Gip{x)x{r), 

hence ()3.17p follows by the dominated convergence theorem and Lemma 3.1, since 

{I + I3){d - a) > d (3.20) 

by jlSl). 

To prove H3.18() we apply the obvious inequality 

^i+/3 _ (j)af^{a -h) for a > 5 > 0, 

and we obtain, by (|3l2ll and (IT7|l . 

/3 



h{T) < (1 + /3)/ / ( f %-uMx)xt{T - u)du 
JW^ Jo \Jo 



X Tr-ufT{x)xT{T — u)du — VT{x,r) jdrdx 

(l + P) [ 1(1 Tr.uMx)XT{T - U)du) ^ 

Jr'^ Jo \Jo J 
X iyj Tr-u(^^Ti-)XTiT - u)vTi-,u) + j-^'^t' ^ i' ^ '^i^ {x)dujdrdx, 
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by H3.3|) . We use 1)3 .71) and boundedness of x to arrive at 

h{T)<C{Ji{T) + ,h{T)), 

where 



Ji{T) = j (^j Tr-u^T{x)du^ (^j Tr-u{^T{-) j Tu-u"fT{-)du'^ {x)diij drdx , 
■hiT) = J Tr-uV{x)duj l^J '^r-uyj Tu-u'^T{-)du j {x)dujdrdx. 



By (2.9) and boundedness of dp (Lemma 3.1) we have 



Ji{T)<CT'^[ [ {G^{x)f+^drdx<CiT~^, 
Jw^ Jo 



by (IT^ and Lemma O 
Similarly, 

UT)<T-^ I {Gip{x)fG{{G^)'+f'){x)dx. 



Hence J2{T) — > 0, since the integral above is finite by 1)3. 2U() and Lemma 13.11 applied three times 
(the function {G<fY^^ satisfies the assumption of the lemma). This finishes the proof of 1)3. 18() . and 
therefore (|3.13() is proved. 

It is not difficult to see that for any <I> G 5(M'^^^), the real random variable {X, <1>) (where X is 
the process in Theorem 2.2 (a)) has a (1 + /?)-stable law with characteristic function 

Sexp{iz(X,^>)} 

= exp|-K^+^|z|i+^ / / |G^'(-,s)(x)|^+'^(l-i(sgnz)(sgn(G^'(-,s)(x)))tan^(l + ^))dsdx 
I Jm'* Jo ^ 2 / 

(3.21) 

(recall that is defined by (|XT|)). 
Now, ()3.13p implies that 

(Xt, «>) ^ (X, as T^oo, (3.22) 

for any <I> G 5(M'^^^). This is almost immediate for > 0, and for general $ one employs weak 
convergence of two-dimensional random variables. See Bojdecki et al (2006a, Lemma 3.4 and Corollary 
3.5) for more details (see also Iscoe, 1986). 

By nuclearity of 5(]R'^+^), (13:22]) imphes Xt ^ X ss,T ^ oo, hence Xt^X. 

i 

Finally, convergence of finite-dimensional distributions of Xt is obtained in exactly the same way 
by virtue of the previous remarks, using ^ of the form 1)3. 8() . 

Next we prove convergence in case (b). 

Again, we prove first convergence To this end we will show that 

i 

lim Eexp{-{XT,^)} = exp!^K2 (^J ^^{x,r)dx^ drj, (3.23) 
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where ^ is defined by and 



K2 = V I pu{x)du\ pi{x)dx. 
Analogously as before, we consider ^ = (p ip, and using (|3.9|) - (|3.12j) we prove (|.S.16|) . (|.S.18|) and 

h{T) - ^i^2 (^^ V{x)dx^ x'^^{r)dr. (3.24) 

In spite of the simpler form of the limit, this case requires a more involved argument. 

Let us start with hiT). Substituiting r' = then u' = Tr' — u, and using (|3.15|1 we write 

l2{T)=Ri{T) + R2iT), (3.25) 

where 

R^{T) = -— / / ru^{x)x{l-r)du] drdx, (3.26) 



1+/3 / rTr \ 1+13- 

Tu(p{x)x{'i- - r)du 





To prove (|3.24|) we show that Ri{T) converges to the desired limit and R2{T) — > 0. 
Note that, by (E^J), 

{G^{x))^+f^dx = oo 



drdx. 
(3.27) 



if (/9 ^ 0, hence we can use the L'Hopital rule to obtain 

"1 / rTr \ 13 



limi?i(r)= limT(l + /3) / / x^+'^(l - ^) / Tuip(x)du] TTMx)rdrdx. (3.28) 
T-.00 T-.00 J^dJ^ VJo / 



After subsitituting n' = ^ and writing 7^ in terms of pu, the expression under the limit on the 
right hand side of (|3.28() has the form 

{! + ())[ I X^^H^-r)( [ [ TrpuTrix - yMy)dydu) [ TrprAx - zMz)dzdxdr. (3.29) 
Jo JR"^ \Jo Jr"^ / JR"^ 

We apply the self-similarity property of the stable density, 

Pau{x) = a~ipuixa~^), (3.30) 
substitute x' = x{Tr)-^ and use (which implies (1 - ^)(1 + (3) = -^). (jlT^ now becomes 

(l + P) / X^^^{'^-r)i / Pu{x - y{Tr)~'^)ip{y)dydu] / pi{x - z{Tr)~^)ip{z)dzdxdr 
Jo JM'* \Jo Jr"^ J Jr"^ 

= (1 + /3) / x'+^(l-r-) / (( [ pudv) *ipTr{x)] Pi*lpTr{x)dxdr, (3.31) 



JR"- \ \J0 



where 

^tiy) =t^^{yt~). (3.32) 



As f^Pudu G L^(R.^) we have 



Pudu] * ifTr 



/ Pudu / ip{x)dx in L^(M'^), 



and the L^-norms are bounded in r G (0, 1]. 
Analogously 

Pi*^Tr^Pi / ^{x)dx in L^^(R^), 

1 

and the L^-" -norms are bounded. By H3.28() this proves that Ri{T) converges to the expression in 

(jHIMl). 

Note that we have also shown that 



1 



T^oo log T 



Tu(p{x)du I dx 



V 



ip{x)dx 



1+/3 



(3.33) 



which will be used later. 

We now turn to R2{T). Note that, by 1)3. 15() . the difference inside [. . .] in ()3.27|) is negative; hence, 
using the elementary inequality 

(a + _ < + (1 + a, 6 > 0, 

(applying it to —[.••]) we have 



l^2(T)| < 



+ 



1 



log T y^d 
i + p 

logT 

Tr 



1 / rTr 



%<p{x) 



\Jo 

1 / rTr 



U 



\J0 



%(p{x) 







Tu(p{x)x [1 — r + — \du\ drdx. 



x(i - - x( 1 - + ^ 



x(i - - X 1 - ^ + 



2 



1+/3 
dti 1 drdx 



du 



i±l 
2 



By the Schwarz inequality applied to the second summand we obtain 



|i?2(r)| < WiT) + (1 + pyw{T)y%(T), 
where hiT) is defined by 1)3. and 



W{T) 



1 



log TyiRdJo \Jq 



1 / rTr 



Tu(p{x) 



X(l - r) - x( 1 - r + - 



1+/3 

dn ) drdx. 



(3.34) 



(3.35) 



Note that 



I2{T)<C-^ [ (rTMx)d 
logT 7,8^70 



!u 1 dx. 



so, by (|3.33l) . I2{T) is bounded. 

By (|3.34|) , to prove that R2 (T) and thus complete the proof of (|3.24|) , it remains to show that 
W{T) 0. 

We use the fact that x is a Lipschitz function and we substitute u' = ^; then 



T' 



W{T) < f f ( f f TpTuix - y)ip{y)udydu] drdx. (3.36) 
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Applying the self-similarity property (|3.3UI) , substituting x' = xT a ^y' = yT a ^ using (|2.f)|) and 
estimating Jq ■ ■ ■ du by Jq ■ ■ ■ du, we obtain 



W{T) < 



C 



I n * (/Jt 1 1 1 i a < 



1 d 1 

Pu{x — y)uTaip(^yTa)dydu\ dx 



logT' 



(3.37) 



where h{x) = pu{x)udu and (pT is defined by (|3.32() . 

We have ||^t||i = f^d (p{x)dx, and it is not difficult to show that h G L^^^{W^) using self-similarity 
oi Pu, the well-known estimate 

C 

P^(x) < ; — ri- — , 

^ - 1 + |a;|^+°' 

and (|2.5j) once again. By (|3.37j) we obtain W(T) ^ as T ^ oo, and (|3.24|) is proved. 

(|3.16() follows by the same estimates as in Bojdecki et al (2005, see (|3.22p . (|3.23p therein). 

Finally, (|3.18j) can be obtained in exactly the same way as in Bojdecki et al (2005, see (|3.1()|) and 
H3.27() - H3.36|) therein). The only difference is that under (|2.5|1 . /(x) = Jq pu{x)du belongs to LP{M.'^) 
for any p < 1 + P (and not to L^~^^ (W^)), so the Young inequality should be applied appropriately, 
and then the corresponding norms of gi,T,92,T (see Bojdecki et al 2005, (3.31)) will have the forms 
C{ip)T^ for e arbitrarily small. This finishes the proof of (|3.23j) . 

Convergence Xt =^y = now follows as in the previous step. One should only observe that 



£;exp{iz(y,$)} 



exp< —K^ 



l+/3|^|l+/3 



^'(x, s)dx 



1+/3 



1-i s, 



ign^z J '^{x,s)dx^^ tan ^{1 + i3)^ds 



Also, convergence of finite-dimensional distributions can be derived similarly as before. The only 
difference is that now (|3.36|) does not hold because x is not Lipschitz. We consider xi''') = l[o,t](^) 
for any fixed t £ [0, 1] (see ()3.8|) ). So, to prove that W(T) defined by (|3.35|) tends to we argue as 
follows. With this form of x we have 



W{T) 



1 



logT 



l-r 



t-r 



TpTu{x - y)ip{y)dydu drdx 



1+13 



By the self-similarity of pu, substituting x' = xT a ^y' = yT ^ ^ and using 1)2. 5|) we obtain 

1 



W{T) 



logT 



l-t+r \ 1+/3 

Pu{x - y)^T{y)dudy] drdx, 



where ifT is given by ()3.32|) . Hence, by the Young inequality. 



W{T) < 



1 



l-t+r 



Pudu 



(remember that ||v?t| 



log T Jo 
i). It suffices to observe that 



< 



Pudu 



r i+z' dr 



1+/3 



dr 



1+/3 



r 1+/3 







dr 



l-t+r \ 1+13 

rpu{x)du I dxdr 



1 \ 
upu{x)du I dx < oo, 
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by the argument following H3.37() . This proves that W(T) 0. So the convergence Xt KiXS^ is 
established. 

□ 

Proof of Proposition 2.4. We consider r = 1, < (5 < 1, < ti < t2 < 1- 

Arguing as in the proof of Proposition 3.3 of Bojdecki et al (2005), it suffices to show that 

P(|(Xt,vp®V)I > <5) < ^(t2-ti) (3.38) 



for any ip E 5(R) such that 

Xit) = ^ i^{s)ds 

satisfies 

< X < l[tut2], (3.39) 

and each ip G 5(M'^), </? > 0. 

Repeating the argument of that proposition (see Bojdecki et al, 2006c, (3.39)-(3.51) ) it is enough 
to prove that 

I <Ciip){t2-h) (3.40) 



and 
where 



II <C{ip){t2-h), (3.41) 

I = [ [ ipTix)xT{T - s) [ %-rMx)XT{T -r)drdsdx, (3.42) 
Jw^ Jo Jo 

II = / / / %-r^T{x)xT{T-r)dr) dsdx, (3.43) 
Jr"^ Jo \Jo J 



and ^t-iXt are given by (|3.15|) . 

We have, after obvious substitutions and by (|3.39|) . 



I < III '/'(^)x(l ~ s)TrT^{x)drdsdx 

Ft Jw^ Jo Jo 



1 



|^(x)|2e-''^l^l"drx(l-s)dsdx. 



where we have used the Plancherel formula and the well known fact that Tu(p{x) = e "1^1" ^(x) 
('"denotes Fourier transform). Hence 



T r 1 



This implies (lOHl) for both d = ^^^^ and d > 



I < I \lp(x)\-'-—dx[\{l-s)ds 

tj, J^d \X\ Jq 
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To prove H3.41() we write 



T 

T 



where 

fx,T{r) = TrTr.V9(x)l[o,i](r), g{r) = x(l - r)l[o,i](r) 
By the Young inequahty we obtain 

Since lls'Hj^l^ < {t2 — ti) by (|3.39j) . it suffices to show that 



sup^ia / Wfx^rWY^^dx <oo. 



This fact follows from Lemma 3.1 in the case d > and from (HOHl) in the case d — — — — 



□ 



4. Comments and questions of interpretation 

In the present and previous papers we have proved limit theorems for occupation time fluctuations 
of (d, a, /3)-branching particle system and described some properties of the limit processes. Some of 
the results raise questions concerning their meaning in terms of properties of the particle system. We 
mention here some of these questions which require further research. 

4.1 Transitions between intermediate and large dimensions 

The results in Bojdecki et al (2005) and the present ones show that in the passage from intermediate 
to large dimensions the spatial structure of the limit process goes from simple (A), corresponding to 
perfect correlation in the case /? = 1, to complicated (truly 5'(M'^)-valued), and the temporal structure 
goes from complicated (long range dependence) to simple (independent increments); at the critical 
dimension both are simple. In subsection 4.4 we comment on the temporal change. We do not have an 
explanation for the spatial change. On the other hand, the size of the fluctuations of the occupation 
time process, measured by Ft, which is typically larger than for the classical central limit theorem in 
the case of long range dependence, does not pass continuously from intermediate to large dimensions 
(as a 01 (3 vary); at the critical dimension they are larger by a logarithmic factor. This type of 
phenomenon is known to occur in some stochastic spatial models. 

4.2 Continuity and jumps 

The main question about the results in this paper is to understand why the (1 + /?)-branching with 
/5 < 1 causes the occupation time fluctuation process to generate jumps in the limit as T — > 00 for the 
critical and large dimensions, but not the intermediate ones. 
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4.3 Poisson vs. equilibrium 

The (d, Q, /3)-branching particle system has equihbrium states in dimensions d> a/f5 (Gorostiza and 
Wakolbinger, 1991), and one may consider the system started off from an equihbrium state instead 
of Poisson (A). Birkner and Zahle (2005) give functional limit theorems with equilibrium and Poisson 
initial states for the fluctuations of the occupation time of the origin of branching random walks on the 
d-dimensional lattice, d > 3. In Bojdecki et al (2004) we did covariance calculations with intermediate 
dimensions for the (d, a, l)-system in equilibrium. On the basis of the results of Birkner and Zahle 
(2005), and Bojdecki et al (2004, 2006a, 2006b), it is natural to expect that for the (d, a, l)-system in 
equilibrium it is possible to prove functional limit theorems for the occupation time fluctuations with 
the same normings Ft as for the initial Poisson condition, and that the results would be as follows. 
For intermediate dimensions, a < d < 2a, the limit would be of the form -fCA^, where ^ is fractional 
Brownian motion. For critical and large dimensions, d > 2q, the results would be the same as the 
corresponding ones with initial Poisson (Bojdecki et al, 2006b). Carrying this further, we think that 
limit theorems can also be proved for the (d, a, /3)-system with /3 < 1 in equilibrium. The normings 
would be the same as for initial Poisson, and the limits would have the same forms as in the Poisson 
case for critical and large dimensions. For intermediate dimensions the limit is expected to be of the 
form KXt], where is a continuous, self-similar, (1 + /3)-stable process with stationary increments and 
long range dependence, which should be a kind of "fractional stable process" . 

Going back to /3 = 1 and intermediate dimensions (covariance calculations in Bojdecki et al, 
2004), in the equilibrium case we get, for the temporal part, fractional Brownian motion (fBm) ^ with 
covariance function 

^{s^ + t'' -\s-t\^), 

where h = 3 — d/a, and in the Poisson case we get sub-fractional Brownian motion (sub-fBm) with 
covariance function 

s^ + t'' -^[{s + tf + \s-t\% 
where h = 3 — d/a again. ^ is defined also for t < 0, and the two processes are related by 

From the functional limit theorem in Bojdcki et al (2006a), and assuming the one for equilibrium, a 
relationship analogous to ()4.1|) would hold for the two corresponding measure-valued limit processes 
(the constant k is the same in both cases). This raises the question of interpreting this mysterious 
result in terms of the particle system. Again going further, we think that for /? < 1 an analogous 
situation would appear. fBm would be replaced by the fractional stable process mentioned above, 
and sub- fBm by the sub-fractional stable process obtained in Bojdeki et al (2005), and they would be 
related analogously as in (|4.1j) . (The sub- fractional stable process with /3 = 1 is sub- fBm, Bojdecki et 
al, 2005; likewise, the fractional stable process with /3 = 1 should be fBm). Then the same question of 
interpretation would arise for the relationship between the equilibrium and Poisson fluctuation limit 
processes with /3 < 1 in terms of the particle system. 

Regarding the model of branching random walks on the lattice (Birkner and Zahle, 2005), modi- 
fying it with the step of the walk in the domain of attraction of a spherically symmetric a-stable law, 
(1 -|- /3)-branching, /3 < 1, and initial Poisson, the results are expected to be analogous to those in 
Bojdecki et al (2005) and the present paper. 
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4.4 Long range dependence vs. independent increments 

Clan recurrence and clan transience have been studied in by Stoeckel and Wakolbinger (1994) for 
d > 3, a = 2, /9 = 1, in equilibrium. A clan is a family of infinitely many particles with eventu- 
ally backwards coalescing paths, the system having started at time — oo. It is shown that for d = 3 
(intermediate dimension) and d = 4 (critical dimension), a.s. all clans visit every fixed ball in 
infinitely often and at arbitrarily large times (this is "clan recurrence"), and for d > 5 (large dimen- 
sions) a.s. each clan visits a fixed ball in in a finite random interval of time and never returns 
(this is "clan transience"). Clan recurrence/transience for super-Brownian motion in equilibrium is 
discussed by Dawson and Perkins (1999). Although clan recurrence/transience has not been studied 
for the branching particle system out of equilibrium (with a = 2, /3 = 1), one may think intuitively 
that the long range dependence of the occupation time fluctuation limit for d = 3 is due to something 
close to clan recurrence: each one of many very large families of related particles visits a fixed ball 
many times, each time adding a random amount to the occupation time of the ball. However, for 
d = 4 this phenomenon still occurs but does not cause long range dependence. On the other hand, 
for d > 5 the independence of increments of the occupation time fluctuation limit (and the classical 
central limit norming T^/^) may be attributed intuitively to something close to clan transience, since 
the large families independently visit a fixed ball only up to finite random times. In order to explore 
these ideas it is necessary to formulate rigorously clan recurrence/transience out of equilibrium (at 
least with initial Poisson), also for the general (d, a, /3)-branching particle system. 

In the case of intermediate dimensions with (3 < 1 (Bojdecki et al 2005), an intriguing question is 
why there are two types of long range dependence regimes, one for P > d/{d + a) and another one for 
(3<d/{d + a). 

4.5 Other long range dependence processes 

For intermediate dimensions and /3 = 1, Poisson initial condition leads to fBm and equilibrium initial 
condition leads to sub-fBm. The question here is if other initial conditions may lead to other long 
range dependence processes, different from fBm and sub-fBm. As a simple example, putting together 
two independent (d, a, l)-branching particle systems, one in equilibrium and the other one with initial 
Poisson (the normings are the same), the occupation time fluctuation limit would be the process 
KX{^ + 0) where ^ is fBm, is sub-fBm, and they are independent (by Bojdecki et al, 2004, the 
constant K is the same in both cases). Other, more interesting, examples will be considered elsewhere. 
The same kind of question arises for /? < 1. A large class of initial conditions (which includes Poisson), 
under which A^^ tends to equilibrium as t ^ oo, is considered by Gorostiza and Wakolbinger (1994). 

Other long range dependence processes may be obtained by incorporating immigration in the 
branching particle systems. Some examples are given by Gorostiza et al (2005) without functional 
convergence proofs. In this case the fluctuations of the rcscaled empirical process itself may have a 
long range dependence Gaussian limit process in dimension d = 1 with a > 1 (see the covariance (1.5) 
in Gorostiza et al, 2005; see also Li and Shiga, 1995, Theorem 1.10, for a superprocess setup). A 
surprising feature of this result is that the fluctuation of the rescaled empirical process is Markovian 
but its limit is not. The Markov property is not necessarily preserved under weak limits, but it would 
be interesting to understand why it is not preserved in this case. 

4.6 Superprocesses and other branching particle systems 

Iscoe (1986) studied the occupation time of the (d, a, /3)-superprocess, which is a measure- valued limit 
of the {d, a, /3)-branching system (see Dawson, 1993, Dawson and Perkins, 1991, Etheridge, 2000, for 
that class of superprocesses). He proved single time limit theorems for the occupation time fluctuations 
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in large dimensions, d > a(l + /3)//9, and also for the intermediate dimension d = 3 in the case q = 2, 
(3 = 1. The limits are 5'(M'^)-valued random variables. For /3 < 1 and large dimensions the limit is an 
»S'(]R'^)-valued (1 + /3)-stable random variable, and a Hilbert subspace of »S'(]R'^) is found where this 
random field lives. (On this point it is relevant to mention that the aim of Perez- Abreu et al, 2005, is 
to determine, for a class of additive processes in the dual of a nuclear Frechet space, a Hilbert subspace 
where such a process lives). For the intermediate dimension d = 3 (a = 2, /3 = 1) an occupation time 
fluctuation limit process is obtained, with fixed ip G »S(R'^), <^ > 0, where the covariance function 
of sub-fBm appears with h = 3/2; that was originally our motivation for investigating long range 
dependence or its absence in occupation time fluctuations of branching systems. 

Occupation time fluctuations for more general branching particle systems, with one and two levels 
of critical branching, have been studied by Dawson et al (2001). The results are single time limits. 
The atypical normings in the occupation time fluctuations for intermediate and critical dimensions 
with one branching level and (3=1 arise as follows. Let Gt^f = /g Tgipds, (p > 0, (p ^ {Gtip ^ oo as 
t ^ oo for transient particle motion). If Gfp grows like a function ft as t ^ oo, then the norming Ft 
for the occupation time fluctuation limit has the form (J^ fsds)^^^. For the standard a-stable process 
on M!^, Gf grows like t^-d/a for a < d < 2a, and like logt for d = 2a. Functional limit theorems for 
the branching systems in Dawson et al (2001) have not been attempted, except in the special cases 
considered in this paper and its predecessors. 
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